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SECTION  1 

introduction 


The  Software  Management  Metrics  presented 
in  this  document  provide  a  top-level  manage¬ 
ment  overview  of  software  development 
status.  These  metrics  are  based  on  Govern¬ 
ment  and  industry  experiences  with  a  pre¬ 
vious  set  of  metrics  known  as  “Software 
Reporting  Metrics"  and  on  an  evaluation 
and  analysis  of  their  use.  The  metrics  can 
provide  cariy  indications  of  potential  soft¬ 
ware  development  problems,  and  can  call 
attention  to  and  stimulate  discussion  leading 
to  early  resolution  of  those  problems.  There 
are  10  metrics  —  5  that  measure  software 
development  progress,  and  S  that  affect  soft¬ 
ware  development  progress.  The  metrics  are 
reported  by  the  contractor  at  each  Program 
Management  Review  (PMR). 

The  successful  use  of  the  metrics  depends 
on  the  program  manager’s  enforcement  of  a 
serious  technical  review  of  the  data  collected 
for  each  metric.  The  metrics  graphs  are  a 
tool  for  escalating  the  discussion  of  impor¬ 
tant  progress  and  status  indicators  to  both 
Government  and  contractor  senior  managers. 

The  metrics  graphs  show  developing  trends 
that  may  indicate  future  problems  and,  when 
analyzed  as  a  set,  can  highlight  inconsisten¬ 


cies  that  might  otherwise  be  overlooked. 
Through  senior  management  reviews  and 
related  communications,  the  significance  of 
the  identified  trends  can  be  determined  and 
appropriate  action  taken. 

Approximately  three  years  of  experience  in 
the  use  and  analysis  of  metrics  has  resulted 
in  this  report,  which  includes  actual  exam¬ 
ples  as  well  as  comments  on  the  behavior 
and  interpretation  of  each  metric. 

This  document  describes  the  Software  Man¬ 
agement  Metrics  and  provides  information 
relating  experience  on  previous  software 
acquisitions.  Section  2  discusses  metrics 
coverage,  reporting,  and  analysis.  Section  3 
describes  the  Software  Management  Metrics; 
each  description  includes  a  statement  of  pur¬ 
pose,  typical  behavior  patterns,  data  inputs, 
tailoring  ideas,  example  plots,  and  interpreta¬ 
tion  notes.  Appendices  A  through  E  provide 
general  information  regarding  software  devel¬ 
opment,  sample  Data  Item  Description  (DID) 
backup  sheets  for  collecting  metrics,  data 
definitions,  and  design  complexity  defini¬ 
tions.  This  document  reflects  the  new  DOD- 
STD-2167A  and  its  associated  DIDs. 
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SECTION  2 


Metrics  Coverage,  Reporting/  and  Analysis 

a 


The  Software  Management  Metrics  described 
in  this  f*  jcument  are  intended  to  assure  that 
(1)  there  are  metrics  to  cover  all  phases  of 
software  development;  (2)  the  metrics  are 
delivered  in  a  manner  that  assures  manage¬ 
ment  visibility  into  important  issues;  and 
(3),  and  most  importantly,  the  metrics 
reported  are  effectively  analyzed  to  identify 
potential  software  development  problems. 
This  section  describes  the  metrics’  coverage 
and  provides  recommendations  regarding 
their  reporting  and  analysis. 


Coverage 

The  metrics  cover  all  phases  of  software 
development.  Figure  1  relates  the  life  of 
each  metric  to  development  milestones.  As 
can  be  seen,  all  development  phases  are 
covered  more  than  once.  This  multiple  cov¬ 
erage  provides  not  only  better  visibility  into 
each  development  phase,  but  also  an  oppor¬ 
tunity  to  verify  the  correctness  of  reported 
data  through  consistency  checks.  The  metrics 
address  two  aspects  of  development:  prog¬ 
ress  and  planning.  The  five  progress  metrics 
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Figure  1.  Metrics  Coverage 
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PLANNING  PROGRESS 


CHANGES  IN 


clearly  indicate  deviations  between  the 
planned  and  actual  status  of  software  devel¬ 
opment.  The  five  planning  metrics  strongly 
influence  software  development  progress. 

Reporting 

The  metrics  should  be  presented  at  each 
PMR  where  they  can  provide  management 
with  visibility  into  potential  cost  and  sched¬ 
ule  impact  problems.  In  order  to  focus  the 
PMR  discussions,  a  pre-PMR  metrics  screen¬ 
ing  is  recommended  that  should  be  accom¬ 
plished  in  two  steps: 

a.  The  contractor  delivers  the  metrics  to  the 
Government  at  least  1  week  prior  to  the 
PMR.  At  that  time,  they  are  discussed  in 
a  Technical  Interchange  Meeting  (TIM) 
or  by  telephone  conference  between  the 


Government’s  and  contractor  s  technical 
staff  to  identify  potential  problem  areas 
for  discussion  at  the  PMR. 

b.  The  results  of  this  TIM  are  discussed 
within  the  System  Program  Office  (SPO) 
at  a  meeting  with  the  SPO  director.  The 
purpose  of  this  meeting  is  to  separate 
issues  to  be  discussed  at  the  PMR  from 
those  to  be, discussed  at  TIMs.  The  rec¬ 
ommendations  arc  conveyed  to  the  con¬ 
tractor,  who  then  tailors  his  PMR 
presentation  accordingly  and  schedules 
any  necessary  TIMs. 

Analysis 

The  metrics  provide  a  mechanism  for  evalu¬ 
ating  the  credibility  of  project  plans  and  for 
identifying  trends  —  not  only  in  deviations 
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Figure  2.  Metrics  Correlation  Matrix 
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between  planned  and  actual  values,  but  also 
in  projections  that  can  be  made  from  actual 
values.  Therefore,  program  managers,  in 
addition  to  their  staffs,  must  understand  if 
not  actively  participate  in  the  metrics'  analy¬ 
sis  in  order  to  draw  conclusions  that  will 
influence  management  decisions.  Two  analy¬ 
sis  techniques  that  have  proven  to  be  effec¬ 
tive  are  correlation  and  extrapolation. 

Correlation 

It  has  been  shown  in  figure  1  that  during 
any  phase  of  development,  several  metrics 
are  being  reported.  There  >s  also  a  strong 
relationship  between  the  reported  metrics; 
that  is,  changes  in  one  metric  should  cause 
changes  in  others.  The  matrix  in  figure  2 
indicates  the  more  common  relationships. 
Changes  in  the  metrics  listed  on  the  left 
should  have  an  impact  on  those  whose 
columns  are  marked  with  an  X.  The  analyst 
should  look  for  inconsistencies  within  the 
related  group  of  metrics.  For  example,  if 
the  Development  Progress  metric  shows  that 


actual  development  is  behind  schedule  and 
the  Personnel  metric  shows  a  reduction  in 
planned  staffing,  then  these  metrics  are  not 
consistent  and  should  be  discussed  with  the 
contractor.  Figure  3  is  an  example  of  just 
such  an  occurrence  on  an  actual  project.  In 
this  case,  the  contractor  acknowledged  the 
discrepancy  and  revised  the  staffing  plan  the 
following  month.  The  correlations  shown  in 
the  matrix  are  not  meant  to  be  all-inclusive. 
Specific  projects  may  have  other  important 
relationships. 

Extrapolation 

One  means  of  analysis  that  can  provide  an 
early  indication  of  potential  problems  is 
extrapolation.  Trends  in  actual  data  can  be 
projected  to  evaluate  their  potential  impact 
on  schedules.  This  applies  to  progress  data 
for  development  items  such  as  Computer 
Software  Unit  (CSU)  design,  code  and  test, 
and  integration,  as  well  as  to  quantitative 
trends  in  software  size,  errors,  and  require¬ 
ments  changes. 
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To  show  how  useful  and  accurate  extrapola¬ 
tion  can  be,  two  examples  are  presented 
from  actual  project  graphs.  These  were 
selected  because  they  contain  enough  data  to 
show  a  trend,  and  in  both  cases  the  trend  is 
distinctly  different  from  the  plan.  Figure  4a 
is  a  graph  taken  from  a  PMR.  Notice  that 
actual  progress  lags  behind  the  plan.  Figure 
4b  shows  an  extrapolation  based  on  actual 
progress  to  date.  This  extrapolation  predicted 
the  development  to  be  completed  around 
week  55,  about  20  weeks  behind  schedule. 

In  figure  4c,  the  actual  data  for  the  balance 
of  the  development  was  obtained  and  plotted. 
The  last  CSU  was  actually  developed  during 
week  55. 


Another  example  is  shown  in  figures  5a 
through  c.  In  this  example,  4  months  elapsed 
between  figures  5b  and  5c.  The  extrapolation 
was  performed  with  only  the  data  shown  in 
5a.  Actual  data  for  the  intervening  4  months 
was  then  obtained  and,  as  can  be  seen,  it 
closely  follows  the  extrapolated  schedule. 
Data  for  the  following  4  months  later  became 
available  and  was  added  to  the  graph.  Notice 
how  the  actual  data  continues  to  follow  the 
extrapolated  schedule  in  spite  of  a  replan  at 
month  42. 

Figure  6  again  illustrates  the  unvarying 
nature  of  an  established  trend.  It  shows  that 
an  extrapolation  made  as  early  as  April  or 
May  would  have  been  quite  accurate. 
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S  E  C  T  I  O  N  3 

The  Metrics 


The  Software  Management  Metrics  are 
described  in  this  section.  Each  metric 
description  includes  a  statement  of  purpose, 
typical  behavior  patterns,  data  inputs,  tailor¬ 
ing  ideas,  plotting  examples,  and  interpreta¬ 
tion  notes.  The  latter  three  require  some 
explanation,  which  is  provided  in  the  follow¬ 
ing  paragraphs. 

Tailor  ns 

Most  of  the  metrics  will  apply  to  all  pro¬ 
grams.  However,  there  are  cases  when 
an  individual  metric  could  be  deleted  or 
replaced  to  meet  specific  needs.  Such  tailor¬ 
ing  also  applies  to  the  individual  data  items 
collected  for  each  metric.  Factors  to  con¬ 
sider  when  tailoring  include  the  nature  of 
the  software  acquisition,  high-risk  areas,  the 
software  metrics  currently  used  by  the  con¬ 
tractor,  the  use  of  more  than  one  program¬ 
ming  language,  the  type  of  testing,  and  the 
number  of  Computer  Software  Configuration 
Items  (CSCIs).  Tailoring  suggestions  are 
included  with  each  metric.  The  result¬ 
ing  set  of  data  input  requirements  for  each 
metric  is  then  contractually  specified  in 
a  DID  referenced  by  the  Contract  Data 
Requirements  List  (CDRL).  Examples  of 
DID  backup  sheets  for  a  basic  set  and  for 
a  tailored  set  of  metrics  are  given  in  appen¬ 
dices  B  and  C,  respectively.  The  reader 
is  cautioned  that  the  tailored  set  is  merely 
an  example,  and  that  each  acquisition 
must  be  analyzed  to  determine  which,  if 
any,  changes  are  needed  and  appropriate. 
Definitions  of  certain  data  items  are  provided 
in  appendix  D. 

Ada  and  Object-Oriented  Design 

The  use  of  Ada  and  the  use  of  an  object- 
oriented  design  methodology  affect  the  col¬ 
lection  of  certain  metrics  data.  Ada  and 
object-oriented  design  may  be  used  sepa¬ 
rately  or  in  conjunction;  therefore,  the  impli¬ 


cations  of  each  are  noted  with  those  metrics 
that  would  require  modification. 

Plotting  Examples 

An  example  plot  is  included  for  each  metric. 
The  plot  format  is  important  and  should 
conform  to  certain  guidelines.  Each  plot 
should  present  at  least  the  past  12  months 
of  planned  and  actual  data  and  the  next  5 
months  of  plan  data.  Review  milestones  are 
indicated  on  the  abscissa.  Plan  data  is  the 
original  plan  submitted  to  the  Government. 
Revised  plans  may  be  added  and  so  labeled, 
but  previous  plans  may  not  be  removed 
because  important  trend  data  would  be  lost. 
The  current  month  is  identified  by  a  bold 
vertical  line  on  the  chart.  The  metrics  are 
normally  reported  at  PMRs  and  therefore 
have  the  same  reporting  period.  The  example 
plots  are  based  on  a  software  development 
initially  estimated  to  require  120,000  Source 
Lines  of  Code  (SLOC)  and  1200  staff 
months  over  a  36-month  schedule.  The  proj¬ 
ect  milestones  are  System  Requirements 
Review  (SRR)  at  month  2,  System  Design 
Review  (SDR)  at  month  5,  Software  Specifi¬ 
cation  Review  (SSR)  at  month  8,  Preliminary 
Design  Review  (PDR)  at  month  12,  Critical 
Design  Review  (CDR)  at  month  17,  Test 
Readiness  Review  (TRR)  at  month  27,  and 
Physical  Configuration  Audit  (PCA)  at 
month  36. 

Interpretation  Notes 

The  interpretation  notes  contain  specific 
guidelines  derived  from  experience  and 
from  analyses  of  actual  projects.  They  are 
intended  to  be  viewed  as  helpful  information, 
not  as  inflexible  principles.  Again,  the  met¬ 
rics  must  be  carefully  analyzed  by  staff  and 
by  senior  managers  if  they  are  to  have  an 
impact  on  management  decisions  and  on 
subsequent  software  development  costs  and 
schedules. 
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Software  Size  Metric 


Purpose 

The  Software  Size  metric  tracks  changes  in 
the  magnitude  of  the  software  development 
effort.  SLOC  to  be  developed  directly  relates 
to  the  software  engineering  effort  necessary 
to  build  the  system.  SLOC  is  also  the  pri¬ 
mary  input  parameter  to  almost  all  software 
cost  estimation  models  used  in  Mission  Criti¬ 
cal  Computer  Resource  (MCCR)  applica¬ 
tions.  An  increase  in  SLOC  estimates  can 
lead  to  schedule  slips  and  staffing  problems. 
An  increasing  trend  should  trigger  steps  to 
counter  the  trend  or  to  plan  for  a  larger 
effort.  SLOC  count  is  initially  an  estimate, 
but  as  the  design  matures  and  code  is  devel¬ 
oped,  the  count  becomes  more  and  more 
accurate  until  it  represents  the  actual  code  at 
completion. 

SLOC  is  usually  tracked  for  each  CSCI  as 
well  as  for  the  total  system.  To  be  complete, 
SLOC  counts  for  all  commercial  off-the- 
shelf  (COTS)  and  modified  off-the-shelf 
(MOTS)  software  should  also  be  included. 

Behavior 

Some  programs  show  increases  in  estimates 
of  SLOC  over  time  while  others  show 
decreases.  Increases  may  be  due  to  a  better 
understanding  of  the  requirements,  a  better 
understanding  of  the  design  implications  and 
complexities,  or  an  optimistic  original  esti¬ 
mate,  whereas  decreases  usually  result  from 
an  overestimate  at  the  beginning  of  the  pro¬ 
gram  and  not  from  changes  in  requirements. 
Both  may  be  due  to  an  original  lack  of 
understanding  and  appreciation  of  the 
requirements. 

Data  Inputs 

Each  reporting  period: 

a.  Estimated  new  SLOC  —  newly  developed 
code. 


b.  Estimated  reused  SLOC  —  existing  code 
used  as  is. 

c.  Estimated  modified  SLOC  —  existing 
code  requiring  change. 

d.  Estimated  total  SLOC  —  all  code  (sum 
of  above). 

A  definition  of  SLOC  that  both  the  contrac¬ 
tor  and  the  SPO  understand  and  accept 
should  be  used.  A  recommended  example 
taken  largely  from  reference  1  is: 

SLOC  includes  each  source  statement  created 
by  project  personnel  and  processed  into 
machine  code.  It  excludes  comments  and 
unmodified  utility  software.  It  includes  job 
control  language,  format  statements,  and 
data  declarations.  It  also  includes  newly 
developed  support  software. 

A  tighter  definition  could  be  developed 
depending  on  the  source  code  language. 

(Note:  An  accepted  measure  of  SLOC  in 
Ada  is  to  count  all  nonlittral  semicolons  (;) 
in  each  package.  SLOC  counts  in  Ada  may 
be  higher  than  with  other  languages  due  to 
the  specificity  and  completeness  of  the 
language.) 

Tailoring  Ideas 

a.  Delete  SLOC  types  not  applicable,  i.e., 
new,  reused,  or  modified. 

b.  Require  separate  data  reporting  for  each 
coding  language  used. 

c.  Require  separate  reporting  for  each  pro¬ 
cessor  and/or  CSCI. 

d.  Report  object  code  size. 

Example  Plot 

Figure  7  illustrates  several  changes  in  coding 
effort  that  would  not  be  shown  if  only  total 
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SLOC  were  reported.  A  month  before  SSR 
it  is  determined  that  some  modified  SLOC 
cannot  be  used  and  that  new  SLOC  will 
have  to  be  developed.  This  results  in  an 
increased  effort,  but  if  only  total  SLOC  were 
reported,  the  increased  effort  would  not  show 
on  the  graph.  It  is  next  determined  that  some 
reused  SLOC  cannot  be  used  and  that  new 
SLOC  will  have  to  be  developed.  Again, 
this  change  requires  additional  effort,  but  if 
only  total  SLOC  is  reported,  it  again  would 
go  unnoticed.  A  month  prior  to  PDR,  the 
example  shows  an  increase  in  new  SLOC 
resulting  from  a  better  understanding  of 
requirements.  This  is  the  only  change 
reflected  in  the  total  SLOC  count. 

Interpretation  Notes 

a.  Software  size  should  not  vary  from  the 
previous  reporting  period  by  more  than 
5  percent  without  a  detailed  explanation 
from  the  contractor  and  related  discus¬ 
sions  regarding  cost  and  schedule 
improvements. 


b.  Changes  in  SLOC  estimates  often  result 
from  a  better  understanding  of  require¬ 
ments,  which  is  desirable.  However, 
increases  in  size  must  be  accounted  for 
in  the  contractor’s  schedule  and  staffing 
plans. 

c.  Total  SLOC  does  not  linearly  relate  to 
effort  because  modified  and  reused  code 
require  increasingly  less  effort  to  develop 
than  new  code.  If  SLOC  to  be  modified 
are  identified  and  counted  at  the  CSU 
level  (including  SLOC  that  must  be 
understood  in  order  to  modify  other 
lines),  then  the  modified  cods  develop¬ 
ment  effort  will  closely  equal  that  for 
newly  developed  code.  Similarly,  reused 
code  may  also  require  coding  effort  to  be 
integrated  into  a  new  system.  Therefore, 
if  the  lines  of  reused  code  requiring  modi 
fication  are  included  in  the  counts  of 
modified  code,  then  the  sum  of  modified 
and  new  code  will  approximate  the  total 
software  development  effort. 


13 


Software  Personnel  Metric 


Purpose 

The  Software  Personnel  metric  tracks  the 
ability  of  the  contractor  to  maintain  planned 
staffing  levels  and  to  maintain  sufficient 
staffing  for  timely  completion  of  the  pro¬ 
gram.  The  software  staff  includes  the  engi¬ 
neering  and  management  personnel  directly 
involved  with  the  software  system  planning, 
requirements  definition,  design,  coding, 
test,  documentation,  configuration  manage¬ 
ment,  and  quality  assurance.  Counts  of 
unplanned  personnel  losses  are  maintained 
so  that  work  force  stability  can  be  tracked. 
Experienced  staff  are  crucial  to  timely  soft¬ 
ware  development.  Experienced  personnel 
are  nominally  defined  as  those  individuals 
with  a  minimum  of  5  years’  experience  in 
MCCR  software  development  and  a  mini¬ 
mum  of  3  years’  experience  in  software 
development  for  applications  similar  to  the 
system  under  development. 

Behavior 

The  planned  staffing  profiles  for  total  soft¬ 
ware  staff  and  for  experienced  software  staff 
should  be  plotted  at  foe  beginning  of  the 
contract.  A  normal  program  may  have  some 
deviations  from  the  plan,  but  the  deviations 
should  not  be  severe.  However,  a  program 
with  too  few  experienced  software  personnel, 
or  one  that  attempts  to  bring  many  personnel 
onboard  during  the  latter  stages  of  the  proj¬ 
ect’s  schedule,  will  most  likely  experience 
difficulty.  The  normal  shape  of  the  total 
software  staff  profile  is  to  grow  through  the 
design  phases,  peak  through  the  coding  and 
testing  phases,  and  then  gradually  taper  off 
as  integration  tests  are  successfully  com¬ 
pleted.  The  shape  of  the  experienced  staff 
profile  should  be  high  during  the  initial  stage 
of  the  project,  dip  slightly  during  CSU 
development,  and  then  grow  somewhat  dur¬ 
ing  testing.  The  ratio  of  total  io  experienced 
personnel  should  typically  be  near  3:1  and 
should  never  exceed  6:1. 


Data  Inputs 

Initial: 

a.  Planned  total  personnel  level  for  each 
month  of  the  contract. 

b.  Planned  experienced  personnel  level  for 
each  month  of  the  contract. 

c.  Expected  attrition  rate. 

Each  reporting  period: 

a.  Total  personnel. 

b.  Experienced  personnel. 

c.  Unplanned  personnel  losses. 

Tailoring  Ideas 

a.  Report  staffing  separately  for  each  devel¬ 
opment  task,  e.g.,  validation  and  veri¬ 
fication  (V&V),  support  software, 
applications  software,  testing,  and 
software  quality  assurance  (SQA). 

b.  Report  staffing  separately  for  special 
development  skills  needed,  e.g.,  Ada, 
database  management  system  (DBMS), 
operating  system,  and  artificial  intelli¬ 
gence  (AI). 

c.  Report  staffing  separately  for  each  devel¬ 
opment  organization. 

Example  Plot 

Figure  8  shows  that  prior  to  CDR  the  actual 
number  of  personnel  war  lagging  behind  the 
plan,  but  that  the  number  of  experienced 
personnel  was  higher  than  planned.  This 
may  indicate  that  the  contractor  was  initially 
having  staffing  problems  and  was  trying  to 
compensate  by  using  additional  experienced 
staff.  Current  levels  rnay  be  appropriate  if 
development  schedules  are  maintained, 
but  the  SPO  should  monitor  this  closely. 
Unplanned  losses  show  a  nominal  trend  and 
do  not  indicate  any  internal  problems.  A 
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Figure  8.  Software  Personnel 


total  of  20  staff,  out  of  an  average  staffing 
level  of  400,  left  the  project  during  the  past 
12  months. 

Interpretation  Notes 

a.  Understaffing  results  in  schedule  slippage 
and,  if  not  corrected,  in  a  continuing  rate 
of  slippage.  Causal  relationships  to  vari¬ 
ous  progress  metrics  should  be  examined. 

b.  Adding  staff  to  a  late  project  will  seldom 


improve  the  schedule  and  often  causes 
further  delays. 

c.  A  program  that  is  experiencing  a  high 
personnel  turnover  rate  cannot  maintain 
needed  continuity.  Losses  that  would 
impair  the  project  knowledge  and  experi¬ 
ence  base  should  be  discussed  with  the 
contractor. 

d.  Initial  staffing  levels  should  be  at  least 
25  percent  of  the  average  staffing  level. 


5 


Software  Volatility  Metric 


Purpose 

The  Software  Volatility  metric  tracks  changes 
in  the  number  of  software  requirements  and 
in  the  contractor’s  understanding  of  these 
requirements.  The  two  graphs  used  for  this 
metric  track  requirements  changes  and  Soft¬ 
ware  Action  Items  (SAIs).  The  graph  of 
requirements  changes  tracks  the  tout'  number 
of  software  requirements  (typically  the  num¬ 
ber  of  “shafts”  in  the  Software  Requirements 
Specifications  (SRSs))  as  weft  as  the  cumula¬ 
tive  number  of  changes  to  those  require¬ 
ments.  The  graph  of  SAIs  tracks  the  number 
of  unresolved  requirement/design  issues. 

Both  graphs  are  good  indicators  of  require¬ 
ment  and  design  stability.  Changes  in  the 
number  of  requirements  (both  additions  and 
deletions)  directly  impact  the  software  devel¬ 
opment  effort.  Changes  are  expected  in  the 
early  stages  as  details  of  the  system’s  opera¬ 
tions  are  being  defined  and  understood.  At 
some  point,  however,  software  requirements 
must  be  frozen.  The  longer  this  takes,  the 
greater  the  impact  on  cost  and  schedule. 

Design  reviews  may  have  several  inconsis¬ 
tencies  between  the  requirements  and  the 
design  or  within  the  design  itself.  When  this 
occurs,  an  SAI  is  opened.  It  may  be  closed 
by  modifying  or  clarifying  the  design  or  by 
modifying  the  requirements.  An  SAI  is 
defined  as  any  discrepancy,  clarification,  or 
requirements  issue  that  must  be  resolved  by 
either  the  contractor  or  the  Government. 

Behavior 

Changes  in  software  requirements  can  be 
expected  to  be  more  numerous  during 
requirements  analysis  and  preliminary  design 
phases.  Changes  occurring  after  CDR  may 
be  expected  to  have  a  significant  schedule 
impact,  even  if  the  change  is  the  deletion  of 
a  requirement.  Therefore,  the  plot  of  cumu¬ 
lative  changes  is  expected  to  rise  more 
steeply  prior  to  PDR  and  show  a  leveling 
off  after  CDR. 


The  plot  of  SAIs  is  expected  to  rise  at  each 
review  and  then  taper  off  exponentially. 
Programs  that  produce  clear  and  complete 
specifications  will  experience  less  of  a  rise 
at  each  review;  and  programs  that  have  good 
communications  among  the  SPO,  the  system 
engineer,  and  the  contractor  will  experience 
a  high  rate  of  decay  to  the  curve. 

Data  Inputs 

Each  reporting  period: 

a.  The  current  total  number  of  requirements. 

b.  The  cumulative  number  of  requirement 
changes  to  include  additions,  deletions, 
and  modifications. 

c.  The  number  of  new  SAIs. 

d.  The  cumulative  number  of  open  SAIs. 

Tailoring  Ideas 

a.  Track  the  longevity  of  open  SAIs,  e.g., 
0-30  days,  30-60  days,  60-90  days,  and 
over  90  days. 

b.  Track  open  SAIs  by  priority. 

Example  Plots 

The  graph  of  requirements  changes  in  figure 
9a  shows  an  upward  trend  in  the  number  of 
changes  prior  to  PDR  and  a  leveling  off 
approaching  CDR.  The  changes  after  PDR 
may  result  in  a  schedule  slip  for  CDR 
because  some  Computer  Software  Compo¬ 
nent  (CSC)  and  CSU  designs  in  the  Software 
Design  Documents  (SDDs)  may  have  to  be 
redone. 

Figure  9b  shows  the  number  of  open  SAIs 
peaking  at  PDR  and  CDR.  The  steady 
decrease  after  each  review  indicates  the  con¬ 
tractor’s  ability  to  resolve  the  issues. 
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Interpretation  Notes 

a.  Requirements  volatility  between  CDR  b.  SAIs  open  more  than  60  days  should  be 

and  TRR  will  result  in  schedule  impacts  closely  examined.  They  could  have  sig 

whose  extent  must  be  determined  through  nificant  schedule  impacts,  especially  if 

discussions  with  the  contractor.  they  have  been  termed  “unimportant.” 


Computer  Resource  Utilization  Metric 


Purpose 

The  Computer  Resource  Utilization  metric 
tracks  changes  in  the  estimated/actual  uti¬ 
lization  of  target  computer  resources  and 
provides  warnings  if  the  limits  of  these 
resources  are  approached.  Three  resources 
typically  monitored  are  CPU  timing,  memory 
(e.g.,  CPU,  mass  storage),  and  I/O  channels 
(e.".,  communications,  bus).  The  system 
architecture  (e.g.,  parallel,  serial,  distributed) 
will  determine  how  the  resources  are  moni¬ 
tored,  but  they  must  be  monitored  to  assure 
that  the  system  will  fit  the  planned  resources. 

Behavior 

Most  projects  experience  an  upward  creep 
in  resource  utilization.  Large  system  acquisi¬ 
tions  typically  specify  a  50  percent  spare 
capacity.  This  means  that  only  one-half  of 
the  capacities  may  be  used,  leaving  50  per¬ 
cent  for  growth.  If  the  utilization  exceeds  50 
percent,  the  project  either  has  to  expand  the 
resource  capabilities  or  change  the  system 
requirements.  Whenever  resource  capabilities 
are  expanded,  the  utilization  curves  affected 
will  drop  to  new  values. 

Dependencies  among  resources  result  in 
parallel  movements.  For  example,  an  expan¬ 
sion  of  memory  not  only  decreases  its  utili¬ 
zation,  but  also  may  allow  the  CPU  to 
operate  more  efficiently,  thus  decreasing 
CPU  utilization.  The  same  memory  expan¬ 
sion  may  also  allow  larger  blocks  of  data  to 
be  transported,  thus  reducing  utilization  of 
the  I/O  channels. 

Data  Inputs 

Initial: 

a.  Planned  spare  for  each  resource. 

Each  reporting  period: 

a.  Estimated/actual  percentage  of  CPU 
utilization. 


b.  Estimated/actual  percentage  of  memory 
utilization. 

c.  Estimated/actual  percentage  of  I/O  chan¬ 
nel  utilization. 

Triloring  Ideas 

a.  Report  combined  utilizations  in  a  multi- 
resource  architecture  that  uses  a  load¬ 
leveling  operating  system. 

b.  Report  utilizations  separately  in  a  multi¬ 
resource  architecture  that  has  dedicated 
functions. 

c.  Report  average  and  worst  case 
utilizations. 

d.  Report  separately  for  development  and 
target  processors. 

e.  Consider  memory  addressing  limits  of 
the  architecture  when  establishing  utiliza¬ 
tion  limits. 

Example  Plot 

A  50  percent  spare  requirement  was  planned 
for  all  three  resources.  Notice  that  each  of 
the  utilizations  plotted  in  figure  10  shows  a 
tendency  to  increase  over  time.  In  this  exam¬ 
ple,  the  CPU  utilization  was  the  first  to 
exceed  the  spare  limit  and  was  corrected 
by  upgrading  to  a  faster  CPU.  If  growth  in 
the  same  computer  series  is  not  possible, 
then  impacts  may  be  felt  on  all  computer 
resources  as  well  as  on  system  and  applica¬ 
tions  software.  It  is  necessary  to  anticipate 
growth  as  early  as  possible  to  minimize  such 
changes. 

Interpretation  Notes 

a.  Performance  deteriorates  quickly  when 
utilization  exceeds  70  percent  for  real¬ 
time  applications. 
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b.  Resource  expansions  should  be  planned 
early  in  the  development  cycle  to  take 
into  account  the  tendency  of  resource 
utilizations  to  increase  over  time. 


c.  Software  development  costs  and  sched¬ 
ules  increase  dramatically  as  computer 
resource  utilization  limits  are  approached 
and  optimization  forces  design  and  coding 
changes. 
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Design  Complexity  Metric 


Purpose 

The  Design  Complexity  metric  tracks  the 
contractor's  ability  to  maintain  an  acceptable 
level  of  complexity  at  the  CSU,  CSC,  and 
CSCI  levels.  A  major  problem  in  software 
development  is  the  creation  of  highly  com¬ 
plex  designs  that  cannot  be  readily  under¬ 
stood,  and  therefore  are  not  rigorously 
verified  through  a  testing  process.  Complex 
designs  are  also  costly  to  maintain.  The 
Design  Complexity  metric  is  a  measure  of 
decision  structure  and  is  calculated  using  a 
method  developed  by  T.  J.  McCabe  (see 
reference  2).  If  design  complexity  is  con¬ 
trolled  so  that  each  CSU,  CSC,  and  CSCI 
can  be  rigorously  tested,  then  system  testing 
becomes  a  low-risk  area.  It  has  been  com¬ 
monly  observed  that  90  percent  of  software 
problems  occur  in  10  percent  of  the  code; 
therefore,  this  metric  tracks  those  CSUs 
witn  the  highest  complexity,  namely,  the  top 
10  percent.  If  a  program  design  language 
(PDL)  is  used,  design  complexity  is  calcu¬ 
lated  directly  from  the  PDL  logic.  Appendix 
E  defines  the  calculations  for  each  complex¬ 
ity  measure. 

Behavior 

A  design  complexity  of  7  is  considered  the 
highest  a  CSU  should  have  without  incurring 
undue  risk.  (The  exception  to  this  is  com¬ 
plexity  resulting  from  the  use  of  a  Case 
statement.)  An  average  complexity  of  7 
would  not  be  excessive  because  only  the  top 
10  percent  are  being  reported.  Because  CSUs 
with  the  highest  complexity  may  be  designed 
later  in  tne  schedule,  this  metric  may  show 
an  increasing  trend  as  CSU  design  pro¬ 
gresses.  This  trend  can  be  correlated  with 
the  CSU  Development  Progress  metric  to 
determine  whether  the  projected  trend  of 
complexity  will  exceed  7  before  all  CSU 
designs  are  completed.  If  so,  steps  can  be 
taken  to  prevent  this.  A  CSC  and  a  CSCI 
should  have  maximum  design  complexities 


j  of  40  and  400,  respectively.  The  behavior 
described  regarding  CSU  complexity  applies 
equally  to  them. 

Data  inputs 

Each  reporting  period: 

a.  Average  design  complexity  of  the  10 
percent  most  complex  CSUs. 

b.  Average  design  complexity  of  the  10 
percent  most  complex  CSCs. 

c.  Design  complexity  of  the  most  complex 
CSCI. 

(Note:  For  object-oriented  designs,  this  met¬ 
ric  will  normally  apply  only  at  the  CSU 
level.  However,  there  are  cases  where  it 
may  apply  :o  CSCs  and  CSCIs  if  the  func¬ 
tions  are  so  organized.) 

Trilorins  Ideas 

a.  Track  reported  top  10  percent  by  cate¬ 
gory;  e.g.,  the  number  of  CSUs  whose 
design  complexity  is  less  than  7,  8  to  20, 
and  over  20. 

b.  Track  design  complexity  of  all  CSCs  by 
category;  e.g.,  number  of  CSCs  whose 
design  complexity  is  less  than  40,  40- 
60,  and  over  60. 

c.  Track  design  com  lexity  of  all  CSCIs. 

d.  Track  code  complexity  after  CDR  using 
Halstead’s  or  McCabe’s  measures  (3,  4], 


Example  Plot 

In  figure  11,  the  plot  of  CSU  design  com¬ 
plexity  shows  an  initial  surge  to  a  complexity 
level  of  9.  The  contractor  then  took  steps  to 
reduce  this  by  dividing  overly  complex  CSUs 
into  two  or  more  CSUs. 
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Figure  11.  Design  Complexity 


Schedule  Progress  Metric 


Purpose 

The  Schedule  Progress  metric  tracks  the 
contractor’s  ability  to  maintain  the  software 
development  schedule  by  tracking  the  deliv¬ 
ery  of  software  work  packages  defined  in 
the  Work  Breakdown  Structure  (WBS).  The 
program's  estimated  schedule  can  be  calcu¬ 
lated  each  month  by  applying  the  relative 
progress  to  date  to  the  program  schedule. 
The  calculation  is  as  follows: 


Estimated 

Schedule 

(months) 


Program  Schedule  (months) 
BCWP/BCWS 


that  had  not  occurred.  The  WBS  for  software 
must  be  tied  to  each  stage  of  testing  so  that 
credit  is  not  given  until  the  adequacy  and 
success  of  tests  at  the  CSU,  CSC.  and  CSCI 
levels  have  been  verified. 

Data  Inputs 

Initial: 

a.  Number  of  months  in  program  schedule. 
Each  reporting  period: 
a.  BCWP  for  software. 


where  BCWP  is  the  budgeted  cost  of  work 
performed  and  BCWS  is  the  budgeted  cost 
of  work  scheduled.  The  use  of  cost  data  to 
estimate  progress  requires  close  coordination 
with  the  costing  staff  to  assure  that  only 
software  costs  are  used  in  this  calculation 
and  that  the  contractor  is  credited  only  for 
work  performed  and  confirmed  by  other 
progress  metrics  such  as  Design,  CSU 
Development,  Testing,  and  Incremental 
Release  Content. 

Behavior 

It  is  not  unusual  for  a  program  to  initially 
fall  behind,  because  insufficient  time  is  usu¬ 
ally  allocated  to  the  design  process.  Thi:. 
trend  should  level  off  as  the  design  is  imple¬ 
mented,  but  may  again  occur  during  testing 
due  to  inadequate  test  planning  and  inade¬ 
quate  testing  at  the  CSU  and  CSC  levels.  It 
is  important  that  testing  at  these  levels  is 
tied  to  WBS  elements  to  assure  visibility 
into  its  adequacy  either  directly  or  through 
a  V&V  function.  Applying  this  metric  retro¬ 
actively  to  certain  procurements  running  50 
to  100  percent  over  schedule  shows  the 
progress  to  be  almost  on  schedule  right  up 
to  system  testing.  This  means  that  credit 
was  given  for  software  development  progress 


b.  BCWS  for  software. 

c.  Number  of  months  in  program  schedule 
(if  revised). 

Tailoring  Ideas 

a.  Track  progress  separately  for  each  CSCI. 

Example  Plot 

At  month  2,  the  plot  in  figure  12  indicates 
that  given  the  current  rate  of  productivity,  it 
will  require  45  instead  of  36  months  to  com¬ 
plete  the  project.  The  ratio  of  work  per¬ 
formed  to  work  scheduled  was  80  percent, 
resulting  in  an  estimated  schedule  of  45 
months  (36/. 8  =  45).  During  succeeding 
months  productivity  dropped  to  around  70 
percent,  resulting  in  an  estimated  schedule 
duration  of  about  51  months.  At  this  point, 
almost  halfway  through  the  original  36- 
month  schedule,  the  schedule  was  revised 
and  extended  to  45  months.  This  change 
does  not  affect  computation  results  because 
increasing  the  schedule  by  25  percent  also 
reduces  the  work  by  20  percent,  so  the  esti¬ 
mated  schedule  months  remain  the  same. 
However,  the  plot  new  indicates  a  slip  of 
only  6  months  versus  15  months  from  the 
original  schedule. 
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Interpretation  Notes 

a.  The  plot  of  this  metric  can  be  used  to 
identify  and  extrapolate  trends.  If  the 
trend  is  up,  it  implies  a  worsening  condi¬ 
tion.  An  extrapolation  can  be  made  to 
predict  the  estimated  schedule  by  extend¬ 
ing  the  extrapolation  until  its  value  inter¬ 
sects  with  the  same  value  on  the  abscissa. 
This  is  the  estimated  schedule  based  on 
the  trend. 


b.  If  the  trend  is  down,  it  indicates  that 
productivity  is  under  control  and  improv¬ 
ing,  and  that  the  overall  schedule  can  be 
predicted  by  extrapolating  this  plot  and 
again  seeking  the  intersection  of  the  plot 
value  with  the  abscissa. 


ORIGINAL  SCHEDULE 


CURRENT 

ESTIMATE 

8888888  REVISED 
SCHEDULE 


Figure  12.  Schedule  Progress 


Design  Progress  Metric 


Purpose 

The  Design  Progress  metric  measures  the 
contractor’s  ability  to  maintain  progress  dur¬ 
ing  the  initial  software  development  phases. 
It  tracks  the  development  of  the  Software 
Requirements  Specifications  (SRSs)  during 
the  system  design  and  software  requirements 
analysis  phases,  and  the  development  of  the 
preliminary  design  portion  of  the  Software 
Design  Documents  (SDDs)  during  the  pre¬ 
liminary  design  phase.  Each  of  these  docu¬ 
ments  culminates  in  a  review  —  Software 
Specification  Review  (SSR)  for  the  SRSs, 
and  Preliminary  Design  Review  (PDR)  for 
the  preliminary  design  portion  of  the  SDDs. 
Tracking  the  progress  of  the  SRSs  and  SDDs 
also  provides  visibility  into  the  contractor’s 
ability  to  hold  SSR  and  PDR  on  schedule. 

Behavior 

System/Segment  Specification  (SSS)  require¬ 
ments  are  initially  allocated  between  hard¬ 
ware,  software,  and  personnel  in  the 
System/Segme'  <t  Design  Document  (SSDD) 
at  System  Design  Review  (SDR).  The  soft¬ 
ware  requirements  in  the  SSDD  are  then 
allocated  to  CSCIs  that  are  each  documented 
in  an  SRS.  Shortly  after  System  Require¬ 
ments  Review  (SRR)  the  contractor  should 
establish  a  schedule  for  developing  each 
SRS.  This  schedule  should  indicate  comple¬ 
tion  1  to  2  months  prior  to  SSR.  Similarly, 
the  SRS  requirements  are  allocated  to  CSCs 
and  documented  in  the  preliminary  design 
version  of  the  SDDs.  This  activity  should 
be  scheduled  for  completion  1  to  2  months 
prior  to  PDR.  Development  of  the  SRSs  and 
the  SDDs  may  fall  behind  the  plan  if  system 
requirements  are  not  clearly  defined.  Clarifi¬ 
cations  may  generate  SAIs  (another  metric) 
whose  open  status  may  also  lead  to  delays 
in  scheduled  reviews. 

Delays  during  these  phases  can  have  both 
positive  and  negative  effects  on  the  balance 


of  the  project  schedule.  If  delays  in  SSR 
and  PDR  result  in  clear  and  precisely  defined 
requirements,  then  improvements  in  the  bal¬ 
ance  of  the  schedule  may  more  than  offset 
those  days.  However,  if  delays  in  SSR  and 
PDR  still  do  not  result  in  well-defined 
requirements,  then  more  and  longer  delays 
can  be  expected  throughout  the  project,  with 
a  corresponding  high  incidence  of  SAIs. 

Data  Inputs 

Initial: 

a.  Number  of  SSDD  software  requirements 
to  be  documented  in  each  SRS  each 
month. 

b.  Number  of  SRS  requirements  to  be  docu¬ 
mented  as  CSCs  in  the  SDDs  each 
month. 

Each  reporting  period: 

a.  Actual  number  of  SSDD  software  require¬ 
ments  completely  documented  in  SRSs. 

b.  Actual  number  of  SRS  requirements  com¬ 
pletely  documented  as  CSCs  in  the 
SDDs. 

Tailoring  Ideas 

a.  Track  the  development  of  the  SDDs  dur¬ 
ing  the  detailed  design  phase;  i.e.,  the 
allocation  of  CSC  requirements  to  CSUs. 
Completion  should  be  scheduled  1  to  2 
months  prior  to  CDR. 

Example  Plot 

In  figure  13  the  actual  number  of  require¬ 
ments  documented  in  the  SRSs  is  shown  to 
fall  behind  the  plan,  but  a  trend  developed 
early  and  provided  a  good  basis  for  planning. 
Based  on  this  slip,  the  SSR  was  delayed  2 
months.  Due  to  the  quality  of  the  SRSs  pro¬ 
duced,  however,  it  was  possible  to  begin 
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development  of  the  SDDs  a  month  ahead  of 
the  rescheduled  SSR  date,  resulting  in  only 
a  1 -month  overall  schedule  slip.  It  also 
appears  that  due  to  the  SRSs’  quality,  the 
SDDs  are  being  developed  at  the  scheduled 
rate,  although  a  month  behind  schedule. 

Interpretation  Notes 

a.  If  the  deviation  between  planned  and 
actual  data  is  increasing,  it  implies  that 
the  further  into  the  requirement  speci¬ 


fication  the  contractor  goes,  the  more 
confusing,  inconsistent,  unclear,  and/or 
untestable  the  requirements  become.  This 
implies  major  problems  with  the  SSDD. 
No  approval  to  proceed  beyond  SSR 
should  be  given  until  a  good  SRS  is  pro¬ 
duced  and  clearly  understood. 

b.  If  the  deviation  between  planned  and 
actual  data  is  decreasing,  it  implies  that 
requirement  problems  may  already  have 
been  addressed  and  that  the  completion 
of  this  task  can  be  projected. 
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Figure  13.  Design  Progress 
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CSU  Development  Progress  Metric 


Purpose 

The  CSU  development  Progress  metric 
tracks  the  contractor’s  ability  to  keep  CSU 
design,  coding,  test,  and  integration  activities 
on  schedule.  After  CDR,  the  next  formal 
milestone  that  indicates  the  status  of  software 
development  is  Test  Readiness  Review 
(TRR).  This  is  too  late  in  a  program  to 
recover  if  significant  problems  are  not  dis¬ 
covered  until  then.  This  metric  provides 
visibility  into  development  progress  through¬ 
out  the  development  phase. 

Behavior 

The  design  of  CSUs  usually  begins  around 
PDR.  A  count  is  plotted  of  the  number  of 
CSU  designs  that  have  passed  internal 
review.  After  CDR,  when  the  contractor 
begins  coding,  a  healthy  program  will  expe¬ 
rience  a  steady  progression  of  CSU  test  com¬ 
pletions.  Sporadic  CSU  development  can  be 
caused  by  factors  such  as  overutilized  devel¬ 
opment  computers  or  under-experienced 
staff.  This  results  in  a  high-pressure  environ¬ 
ment  in  which  all  software  becomes  due  at 
once.  For  a  well-run  program,  the  plots  of 
CSUs  designed,  tested,  and  integrated  should 
each  rise  with  a  fairly  constant  slope, 

The  CSU  design,  test,  and  integration  sched¬ 
ule  should  be  reported  by  the  contractor  at 
CDR.  The  number  of  CSUs  whose  design 
packages  have  been  closed,  the  number  of 
CSUs  passing  CSU  test  and  placed  under 
configuration  control,  and  the  number  of 
CSUs  integrated  will  be  updated  at  the  end 
of  each  calendar  month  of  the  contract. 

Data  Inputs 

Initial: 

a.  Number  of  CSUs  to  be  designed  each 
month. 

b.  Number  of  CSUs  tc  be  tested  each 
month. 


c.  Number  of  CSUs  to  be  integrated  each 
month. 

Each  reporting  period: 

a.  Actual  number  of  CSUs  designed. 

b.  Actual  number  of  CSUs  tested. 

c.  Actual  number  of  CSUs  integrated. 

(Note:  Ada  CSU  design  progress  may  be 
measured  by  counting  the  number  of  pack¬ 
ages  whose  specifications  are  complete.  This 
is  using  Ada  as  a  PDL  and  measuring  CSU 
(package)  design  progress  to  completion. 

For  object-oriented  designs,  CSU  integration 
progress  may  be  tied  to  an  integration  plan 
rather  than  to  CSCs.) 

Tailoring  Ideas 

a.  Plot  very  large  CSCIs  separately  or  plot 
each  CSCI  when  there  are  only  two  or 
three  in  the  system. 

b.  Track  CSU  development  progress  by 
CSCs. 

c.  Track  number  of  CSUs  for  which  PDL 
code  has  been  developed. 

d.  Report  defect  density  from  errors  found 
at  CSU  design  inspections,  i.e.,  average 
number  of  defects  per  CSU. 

e.  Report  defect  density  from  errors  found 
at  CSU  test  inspections,  i.e.,  average 
number  of  defects  per  CSU. 

Example  Plot 

In  figure  14,  the  plot  of  CSUs  actually 
designed  follows  closely  the  plot  of  CSUs 
planned  to  be  designed  until  3  months  after 
PDR.  At  this  point,  the  actual  number  begins 
to  deviate  significantly  from  the  planned 
number.  The  contractor’s  explanation  was 
that  changing  requirements  prolonged  the 
design  of  the  sy stem. 


26 


1000 


3 

8 


PDR 


CDR 


TRR 


Figure  14.  CSU  Development  Progress 


The  number  of  CSUs  actually  tested  and 
integrated  lags  behind  the  plan.  The  SPO 
ascettained  from  the  contractor  that  the  lag 
resulted  from  an  increased  number  of  lines 
of  code  to  be  developed,  tested,  and  inte¬ 
grated.  These  slips  may  indicate  that  the 
project  schedule  cannot  be  met.  The  SPO 
should  closely  monitor  the  progress  of  testing 
and  integration  to  effect  any  possible  sched¬ 
ule  improvements  with  the  contractor. 

Interpretation  Notes 

Wide  variations  in  schedule  profiles  have 
been  followed  by  different  projects.  The 
following  general  information  should  not  be 
misused  through  overly  specific  interpreta¬ 
tion.  It  is  offeied  on  the  premise  that  some 
guidance  is  better  than  none.  For  more  data, 
see  references  1  and  5.  Contributors  to  wide 


variations  in  schedule  performance  are  the 
availability  and  use  of  modem  software 
development  tools  such  as  PDLs,  automated 
debugging  facilities,  and  database  tools. 

a.  Schedule  for  Software  Development 

The  number  of  calendar  months  (M)  from 
the  beginning  of  software  design  to  the 
end  of  PCA  is  related  to  the  number  of 
software  development  staff  months  (SM) 
necessary  to  complete  the  software. 
(Complete  software  refers  to  software 
that  has  been  designed,  coded,  tested, 
integrated,  system  tested,  and  docu¬ 
mented.)  The  following  formulas  have 
been  empirically  derived  from  a  database 
of  ESD/MITRE  projects.  The  formulas 
describe  curves  for  three  project  percentile 
values.  For  example,  only  5  percent  of 
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the  projects  in  the  Construction  Cost 
Model  (COCOMO)  database  were  able 
to  preserve  a  schedule  equal  to  or  shorter 
than  the  schedule  predicted  by  M  = 
3*VSM. 

Percentile 

M=3*VSM  5% 

M=3.8*VSM  50% 

M=4.6*VSM  95% 

b.  Source  Lines  of  Code  per  Staff  Month 
Typical  SLOC/SM  values  are: 

150  for  easy  code 

70  for  moderately  difficult  code  (C3I 
system) 

30  for  difficult  code  (e.g.,  security) 

(Note:  SM  refers  to  the  number  of  software 


staff  months  expended  from  contract  award 
through  PC  A.) 

c.  Allocation  of  Staff  and  Schedule  for  Soft¬ 
ware  Development 

These  are  desired  values  based  on  experi¬ 
ence  with  Mission  Critical  Computer 
Resource  (MCCR)  software  programs 
similar  to  the  C3I  systems  developed  at 
ESD.  Wide  variations  from  these  values 
have  been  observed,  but  experience  indi¬ 
cates  that  increases  in  design  effort  lead 
to  reduced  integration  and  test  effort  and 
to  higher  quality  products. 


Staff  Months  Schedule 
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Testing  Progress  Metric 


Purpose 

The  Testing  Progress  metric  measures  the 
contractor’s  ability  to  maintain  testing  prog¬ 
ress  and  the  degree  to  which  the  software  is 
meeting  system  requirements.  Two  separate 
graphs  are  used  for  this  metric.  The  test 
status  graph  tracks  the  progress  of  CSCI  and 
system  testing  against  a  plan.  A  graph  of 
Software  Problem  Reports  (SPRs)  focuses 
attention  on  the  number  of  open  software 
problems  and  the  density  of  discovered 
errors. 

The  test  status  graph  shows  separate  plots 
for  CSCI  formal  qualification  tests  and  for 
system  tests.  CSCI  testing  begins  after  TRR, 
and  system  testn '  begins  at  the  conclusion 
of  CSCI  testing  and  is  completed  prior  to 
PCA. 

The  SPR  graph  plots  the  number  of  new 
SPRs  reported  each  month  and  the  current 
number  of  open  SPRs.  Also  plotted  is  SPR 
density,  which  is  the  cu  ..ulative  number  of 
SPRs  per  1000  SLOC.  A  separate  set  of 
SPR  plots  is  shown  for  problems  identified 
during  CSCI  and  «v;  n  testing. 

Behavior 

The  test  status  graph  snows  the  planned  and 
actual  counts  of  CSCI  and  system  tests.  The 
plot  of  tests  completed  sh  jld  overlay  the 
plot  of  tests  scheduled  if  ad  tests  are  passed 
on  schedule.  However,  st  programs  expe¬ 
rience  schedule  slips  or  failed  tests.  There¬ 
fore,  the  plot  of  tests  completed  can  be 
expected  to  fall  below  the  plot  of  tests  sched¬ 
uled.  The  extent  of  this  difference  and  the 
trend  it  proscribes  are  indicators  of  the  readi¬ 
ness  of  the  CSCIs  and/or  the  system  for 
testing  and  of  the  time  it  will  take  to  com¬ 
plete  testing. 

The  SPR  graph  provides  a  more  detailed 
view  of  testing  progress.  If  the  number 


of  open  SPRs  from  CSC  testing  is  not 
approaching  zero  as  TRR  approaches  (or  as 
PCA  approaches  for  CSCI  and  system  test¬ 
ing),  then  these  milestones  will  have  to  be 
delayed.  Schedule  slips  may  be  estimated  by 
observing  the  trend  of  open  SPRs.  An  ambi¬ 
tious  test  plan  may  prevent  the  number  of 
open  problems  from  decreasing  until  all  tests 
have  been  performed.  This  is  to  say  that 
new  problems  may  be  generated  faster  than 
old  ones  can  be  resolved. 

Data  Inputs 

Initial: 

a.  Planned  number  of  CSCI  tests  to  be  com¬ 
pleted  during  each  month  of  the  contract. 

b.  Planned  number  of  system  tests  to  be 
completed  during  each  month  of  the 
contract. 

Each  reporting  period: 

a.  Number  of  CSCI  tests  passed. 

b.  Number  of  system  tests  completed. 

c.  Number  of  new  SPRs. 

d.  Cumulative  number  of  open  SPRs. 

e.  SPR  density  :  cumulative  number  of  SPRs 
per  1000  SLOC. 

(Note:  If  the  design  is  object-oriented  and/or 
is  not  organized  by  CSCs  and  CSCIs,  then 
testing  progress  must  be  tracked  against 
functionality  tests  described  in  both  the  inte¬ 
gration  and  system  test  plans.) 

Tailoring  Ideas 

a.  Track  the  longevity  of  open  SPRs.  For 
example,  track  the  number  open  0-30 
days,  30-60  days,  60-90  days,  and  over 
90  days. 
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b.  Track  open  SPRs  by  priority,  e.g.,  high, 
medium,  and  low;  or  critical  for  integra¬ 
tion,  critical  for  initial  operation,  or  other. 

c.  Track  open  SPRs  by  type  of  software 
affected,  e.g.,  application,  operating  sys¬ 
tem,  and  support  test. 

d.  Report  SPRs  by  open,  assigned,  pending, 
resolved,  rejected,  and/or  closed. 

e.  Report  SPR  density  by  category;  e.g., 
the  number  of  CSUs  whose  SPR  density 
is  0-10,  1 1-20,  21-30,  and  over  30. 

Example  Plots 

In  the  example  test  status  graph  shown  in 

figure  15a,  the  plot  of  tests  completed  falls 


behind  those  scheduled.  It  can  be  expected 
that  this  trend  will  continue  unless  the  con¬ 
tractor  can  increase  the  rate  of  testing  by 
increasing  the  number  of  qualified  personnel 
conducting  tests  or  the  number  of  hours 
spent  each  day  performing  tests,  and  neither 
of  these  is  likely.  This  means  that  the  start 
of  system  testing  will  be  delayed  and  that  its 
rate  of  progress  will  probably  be  similar  to 
that  for  CSCI  testing. 

The  SPR  graph  in  figure  15b  shows  a  sepa¬ 
rate  plot  for  SPRs  occurring  prior  to  CSCI 
and  system  testing.  The  TRR  was  resched¬ 
uled  because  there  were  too  many  open 
SPRs.  The  SPR  density  ranges  between  8 
and  20  SPRs/ 1000  SLOC,  which  is  within 
the  normal  range. 


TRR  PCA 


Figure  15a.  Testing  Progress/Test  Status 


Figure  15b.  Testing  Progress/Software  Problem  Reports 


Interpretation  Notes 

a.  The  number  of  SPRs/1000  SLOC  is  an 
indication  of  testing  adequacy  and  of 
code  quality.  A  normal  range  may  be 
between  5  and  30  SPRs/1000  SLOC, 
with  10  to  20  a  safer  range.  Too  few 
SPRs  may  indicate  poor  testing  and  too 
many  may  indicate  poor  code  quality.  In 


either  case,  the  code  may  still  contain  a 
large  number  of  undetected  errors. 

b.  If  the  plot  of  open  SPRs  has  a  positive 
slope,  it  indicates  that  problems  are 
being  identified  faster  than  they  can  be 
resolved.  If  the  slope  is  negative,  then 
the  resolution  of  problems  can  be 
projected. 
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Incremental  Release  Content  Metric 


Purpose 

The  Incremental  Release  Content  metric 
tracks  the  schedule  and  the  number  of  CSUs 
per  release  in  order  to  monitor  the  contrac¬ 
tor’s  ability  to  preserve  schedule  and  func¬ 
tionality  in  each  planned  release.  This  metric 
also  applies  to  software  builds.  A  common 
approach  used  to  preserve  schedule  is  to 
postpone  functional  capability.  Decreases  in 
the  number  of  CSUs  per  release  may  indicate 
an  off-loading  of  functions  from  early 
releases  to  later  releases.  Increases  in  the 
number  of  CSUs  per  release  may  indicate 
that  a  program  is  having  unanticipated 
growth  in  the  complexity  of  the  functions  to 
be  delivered. 


Behavior 


Ideal  behavior  is  for  the  number  of  CSUs  in 
each  release  and  the  schedule  dates  to  remain 
unchanged.  However,  typical  behavior  is  for 
the  number  of  CSUs  in  early  releases  to 
decrease  and  the  number  in  later  releases  to 
increase  as  the  release  dates  approach.  This 
postponement  is  exacerbated  by  code 
growth,  which  also  increases  the  number  of 
CSUs  in  later  releases.  In  a  program  whose 
designs  were  prepared  accurately  and  com¬ 
pletely  for  PDR  and  CDR,  the  number  of 
CSUs  should  not  increase  significantly  after 
CDR.  A  program  being  developed  on  sched¬ 
ule  will  implement  the  planned  number  of 
CSUs  in  each  release. 


The  planned  number  of  CSUs  to  be  included 
in  each  anticipated  release  is  updated  at  the 
end  of  each  month  of  the  contract,  but  the 
original  plan  always  remains  on  the  graph. 
The  original  plan  for  each  release  extends  to 
the  release  date. 


(Note:  The  term  “release”  does  not  necessar¬ 
ily  refer  to  contractually  obligated  delivery 
of  software.  The  normal  software  develop¬ 
ment  process,  in  which  the  contractor  assem¬ 


bles  increasing  portions  of  the  software, 
results  in  releases  that  may  be  monitored. 
As  each  of  these  is  prepared,  the  functions 
expected  can  be  compared  to  the  functions 
actually  included.) 


Data  Inputs 

Initial: 


a.  Planned  number  of  CSUs  to  be  included 
in  each  release. 


b.  Planned  completion  date  for  each  release. 
Each  reporting  period: 


a.  Current  number  of  CSUs  to  be  included 
in  each  release. 


b.  Current  completion  date  for  each  release. 


Tailoring  Ideas 

a.  Track  the  number  of  CSCs  in  each  release 
as  another  measure  of  the  functionality 
included  in  each  release. 


b.  Track  the  number  of  requirements  satis¬ 
fied  in  each  release. 


Example  Plot 

Figure  16  shows  content  and  schedule:  a 
two-dimensional  tracking  of  planned 
releases.  Shortly  after  CDR  the  number  of 
CSUs  included  in  release  1  is  reduced  from 
200  to  approximately  150.  This  means  that 
250  CSUs  will  have  to  be  integrated  for 
release  2;  not  the  200  originally  planned. 
The  number  of  CSUs  in  release  2  is  reduced 
the  following  month  from  400  to  350, 
thereby  bringing  the  increment  between  the 
two  builds  back  to  the  200  CSUs  originally 
planned.  In  the  meantime,  the  number  of 
CSUs  in  release  3  has  been  increasing  due 
to  an  increase  in  total  software  size.  At  the 
point  where  release  2 A  begins,  the  number 
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of  CSUs  to  be  integrated  for  release  3  has 
grown  to  almost  500.  It  is  obvious  to  the 
contractor  that  an  additional  release  is 
needed;  hence,  release  2A  is  inserted,  wh'ch 
integrates  300  of  the  500  CSUs,  leaving 
approximately  200  for  release  3. 

Another  solution  not  shown  in  figure  16 
would  be  to  reschedule  the  completion  dates 
for  each  release.  However,  this  is  less 
acceptable  to  both  the  contractor  and  the 
Government  because  each  schedule  slip 
attracts  management's  attention.  In  this 
example,  however,  the  productivity  rates 
implied  by  the  progress  to  date  indicate  that 


the  scheduled  completion  dates  for  releases 
2A  and  3  will  likely  be  slipped.  This  likeli¬ 
hood  should  be  discussed  with  the  contractor. 

Interpretation  Notes 

a.  The  number  of  CSUs  per  release  should 
remain  reasonably  stable.  However,  a 
small  increase  in  the  number  of  CSUs 
can  be  expected  as  the  program's  design 
matures. 

b.  Builds  should  be  encouraged  that  corre¬ 
spond  to  operationally  useful  capabilities 
as  soon  as  it  is  practicable. 


CDR 


Figure  16.  Incremental  Release  Content 
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APPENDIX  A 


General  Software  Acquisition  Information 


The  metrics  described  in  Section  3  are 
accompanied  by  notes  that  relate  to  each 
metric.  This  appendix  contains  phase-specific 
information  applicable  to  many  software 
acquisitions.  The  information  was  obtained 
from  software  acquisition  personnel  and 
from  references  1  and  5,  and  is  intended  to 
provide  general  guidance.  Exceptions  to 
these  guidelines  may  be  appropriate  for  spe¬ 
cific  programs;  but  when  an  exception  is 
made,  it  is  important  to  have  a  clear  under¬ 
standing  of  the  justification. 

1.  Pre-Award 

a.  Resource  Availability 

Software  development  resources  should 
be  evaluated  in  a  manner  similar  to  hard¬ 
ware  manufacturing  facilities: 

(1)  Large  software  development  projects 
should  use  modem  methodologies  with 
automated  support  tools  for  software 
development.  Tools  should  be  available 
for  such  functions  as  static  analysis,  test¬ 
ing,  design,  measurements,  and  configu- 

'  ration  management. 

(2)  Sources  of  experienced  software  engi¬ 
neering  personnel  must  be  identified  and 
available  during  the  designated  software 
development  time  period. 

b.  Requirements  Definition 

System  requirements  must  not  be  con¬ 
fused  with  design  or  implementation 
details  or  vice  versa. 

c.  Contract  Preparation 

The  Instructions  for  Proposal  Preparation 
(IFPP),  Statement  of  Work  (SOW),  and 
CDRL  should  include  requirements  for 
the  contractor  to  deliver  the  Software 
Management  Metrics  graphs  and  data. 


2.  Source  Selection 

a.  Difficult  and  high-risk  functions  should 
receive  early  and  adequate  attention. 
Workable  solutions  should  be  demon¬ 
strated  before  start  of  full-scale 
development. 

b.  The  software  development  schedule  must 
accommodate  the  practicalities  of  the 
process.  It  should  not  be  “success  ori¬ 
ented.”  Schedule  compression  adds 
greatly  to  cost  and  does  not  generally 
produce  an  earlier  product. 

c.  The  difficulty  of  a  project  increases  as 
more  organizations.  Government  as  well 
as  contractor,  participate.  The  Govern¬ 
ment  should  avoid  becoming  an  interface 
between  two  or  more  contractors. 

d.  The  Government  should  assure  that  any 
software  development  by  subcontractors 
is  closely  managed  by  the  prime  con¬ 
tractor.  This  is  best  accomplished  by 
collocation. 

e.  Large  development  projects  should  have 
functional  capabilities  implemented  in 
phased  releases.  If  schedule  problems 
develop,  useful  capabilities  can  be  pro¬ 
vided  before  the  development  program  is 
completed. 

3.  Preliminary  and  Critical  Design 
Reviews 

a.  Development  phases  should  not  be 
allowed  to  proceed  faster  than  the 
achieved  ground  work  can  support.  For 
example,  the  design  should  not  advance 
beyond  the  status  of  requirements  defini¬ 
tion,  coding  beyond  what  has  been 
designed,  and  testing  beyond  the  stability 
of  the  product. 

b.  Monthly  or  bimonthly  software  TIMs 
should  supplement  the  formal  reviews. 
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Programs  that  are  large  or  on  tight  sched¬ 
ules  should  have  in-plant  Government 
technical  representatives  so  that  more 
frequent  contact  is  maintained  between 
the  developers  and  the  acquisition  agency. 

4.  Test  and  Integration 

a.  Software  should  be  tested  by  a  separate 
and  independent  organization  from  that 
which  developed  the  software. 


b.  The  formal  software  acceptance  testing 
schedule  should  allow  time  and  access  to 
the  equipment  for  Government  personnel 
to  exercise  the  system. 

c.  Integration  test  planning  should  be  com¬ 
pleted  early  enough  to  ensure  that  test 
issues  can  influence  design,  and  allow 
long-lead-time  test  facilities  to  be 
acquired.  For  most  programs,  test  plan¬ 
ning  should  start  before  PDR. 
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APPENDIX  B 

Sample  DID  Backup  Sheet 

Possible  uses: 

DI-A-7089  Conference  Minutes 
DI-MGMT-80227/T  Contractor  Progress, 
Status,  and  Management  Report 


The  contractor  shall  substitute  or  add  the 
following  to  Blocks  3  and  10  of  the  DID: 

Block  3:  to  provide  the  USAF  with  insight 
into  the  software  development  progress, 
status,  and  problem  areas. 

Block  10:  replace  section  10  with  the 
following: 

The  Software  Management  Metrics  are  a 
high-level  summary  of  the  status  of  the  soft¬ 
ware  development  effort.  Specific  guidelines 
for  their  contents  are  given  in  the  paragraphs 
below. 

In  the  context  of  this  DID,  the  term  “current 
reporting  period”  shall  refer  to  the  interval 
of  time  between  the  past  (i.e.,  most  recent) 
and  present  submission  dates.  The  frequency 
of  submission  for  the  report  is  provided  in 
the  CDRL  item  for  this  DID. 

The  contractor  shall  provide  data  for  the 
metrics  listed  below  in  both  graphic  and 
tabular  form.  This  information  shall  be 
updated  as  necessary  for  the  current  reporting 
period.  The  metrics  data  shall  be  provided 
in  graphics  form  at  the  contract  level  for 
each  contractor/subcontractor.  Graphs  shall 
also  be  prepared  at  the  CSCI  level  for  certain 
metrics  as  designated  below.  Additional 
graphs  shall  be  provided  as  required  to 
address  high-risk  or  problem  CSCIs.  Graphs 
show  12  months  of  history  and  5  months  of 
projections  where  appropriate.  Revised  plans 
may  be  added  to  the  graphs,  but  previous 
plans  including  the  original  plan  shall  not 
be  removed. 

Software  Size 

For  the  current  reporting  period,  provide  in 


tabular  and  graphic  form  for  each  CSCI  an 
estimate  of: 

1 .  Newly  developed  Source  Lines  of  Code 
(SLOC); 

2.  Reused  SLOC  —  existing  code  used  as 
is; 

3.  Modified  SLOC  —  existing  code  requir¬ 
ing  change;  and 

4.  Total  SLOC  —  all  code  (sum  of  above). 

SLOC  includes  each  source  statement  created 
by  project  personnel  and  processed  into 
machine  code.  It  excludes  comments  and 
unmodified  utility  software.  It  includes  job 
control  language,  format  statements,  and 
data  declarations.  It  also  includes  newly 
developed  support  software. 

Software  Personnel 

Provide  a  plan  that  includes,  for  each  month 
of  the  contract: 

1 .  The  planned  number  of  software  person¬ 
nel,  and 

2.  The  planned  number  of  experienced  soft¬ 
ware  personnel. 

For  the  current  reporting  period,  provide 
actual  counts  of: 

1 .  The  total  number  of  software  personnel, 

2.  The  number  of  experienced  personnel, 
and 

3.  The  number  of  unplanned  personnel 
losses. 

Software  personnel  include  engineering  and 
management  personnel  directly  involved 
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with  software  system  planning,  requirements 
definition,  design,  coding,  test,  integration, 
documentation,  configuration  management, 
and  quality  assurance.  Experienced  personnel 
are  defined  as  those  with  over  5  years  of 
experience,  of  which  3  years  are  with  sys¬ 
tems  similar  to  the  one  under  development. 

Software  Volatility 

For  the  current  reporting  period,  provide: 

1 .  The  current  total  number  of  software 
requirements, 

2.  The  cumulative  number  of  requirements 
changes, 

3.  The  number  of  new  Software  Action 
Items  (SAIs),  and 

4.  The  cumulative  number  of  open  SAIs. 

An  SAI  is  defined  as  any  discrepancy,  clari¬ 
fication  item,  or  requirement  issue  that  must 
be  resolved  by  either  the  contractor  or  the 
Government. 

Computer  Resource  Utilization 

For  the  current  reporting  period,  provide 
estimated  percentage  utilizations  of: 

1 .  CPU  computation  power  for  each  CPU 
in  multi-CPU  configurations, 

2.  On-line  memory,  and 

3.  I/O  channel  capacity. 

Design  Complexity 

For  the  current  reporting  period,  provide: 

1 .  The  average  design  complexity  of  the  10 
percent  most  complex  CSUs, 

2.  The  average  design  complexity  of  the  10 
percent  most  complex  CSCs,  and 

3.  The  design  complexity  of  the  most  com¬ 
plex  CSCI. 

(Note:  See  attached  for  definition  and 
discussion  of  design  complexities.  (This 
discussion  is  contained  in  appendix  E  of  this 
document.)) 


Schedule  Progress 

Initially  provide  the  number  of  months  in 
the  program  schedule. 

For  the  current  reporting  period,  provide: 

1 .  The  budgeted  cost  of  work  performed 
(BCWP)  for  software, 

2.  The  budgeted  cost  of  work  scheduled 
(BCWS)  for  software,  and 

3.  The  number  of  months  in  the  program 
schedule  (if  revised). 

Design  Progress 

Provide  a  plan  that  includes,  for  each  month 
of  the  contract: 

1 .  The  number  of  System/Segment  Design 
Document  (SSDD)  software  requirements 
to  be  allocated  and  documented  in  Soft¬ 
ware  Requirement  Specifications  (SRSs) 
each  month,  and 

2.  The  number  of  SRS  requirements  to  be 
allocated  and  documented  as  CSCs  in 
Software  Design  Documents  (SDDs) 
each  month. 

For  the  current  reporting  period,  provide: 

1 .  The  number  of  SSDD  software  require¬ 
ments  completely  documented  in  SRSs, 
and 

2.  The  number  of  SRS  requirements  com¬ 
pletely  documented  as  CSCs  in  SDDs. 

CSU  Development  Progress 

Provide  a  plan  that  includes,  for  each  month 
of  the  contract: 

1.  The  number  of  CSUs  to  be  designed, 

2.  The  number  of  CSUs  to  be  tested,  and 

3.  The  number  of  CSUs  to  be  integrated. 

For  the  current  reporting  period,  provide 
actual  counts  of: 

1 .  The  number  of  CSUs  designed, 

2.  The  number  of  CSUs  tested,  and 

3.  The  number  of  CSUs  integrated. 
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Testing  Progress 

Provide  a  test  plan  that  includes,  for  each 

month  of  the  contract: 

1 .  The  number  of  CSC  tests, 

2.  The  number  of  CSCI  tests,  and 

3.  The  number  of  system  tests  scheduled  to 
be  performed. 

For  the  current  reporting  period,  provide: 

1.  The  cumulative  number  of  CSC,  CSCI, 
and  system  tests  passed; 

2.  The  number  of  new  software  problem 
reports  (SPRs); 

3.  The  cumulative  number  of  open  SPRs; 
and 


4.  The  cumulative  number  of  SPRs  per  1000 
SLOC. 

Incremental  Release  Content 

Provide  a  plan  that  includes: 

1.  The  number  of  CSUs  to  be  included  in 
each  incremental  build/release,  and 

2.  The  completion  date  for  each  incremental 
build/release. 

For  the  current  reporting  period,  provide: 

1 .  The  number  of  CSUs  that  were  added  to 
or  deleted  from  each  incremental  build/ 
release,  and 

2.  Any  changes  in  completion  dates  for 
each  incremental  build/release. 
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APPENDIX  C 

Enhanced  DID  Backup  Sheet 

Possible  uses: 

DI-A-7089  Conference  Minutes 
DI-MGMT -80227/T  Contractor’s  Progress, 
Status,  and  Management  Report 


The  contractor  shall  substitute  or  add  the 
following  to  Blocks  3  and  10  of  the  DID: 

Block  3:  to  provide  the  USAF  with  insight 
into  the  software  development  progress, 
status,  and  problem  areas. 

Block  10:  replace  section  10  with  the 
following: 

The  Software  Management  Metrics  are  a 
high-level  summary  of  the  status  of  the  soft¬ 
ware  development  effort.  Specific  guidelines 
for  their  contents  are  given  in  the  paragraphs 
below. 

In  the  context  of  this  DID,  the  term  “current 
reporting  period”  shall  refer  to  the  interval 
of  time  between  the  past  (i.e.,  most  recent) 
and  present  submission  dates.  The  frequency 
of  submission  for  the  report  is  provided  in 
the  CDRL  item  for  this  DID. 

The  contractor  shall  provide  data  for  the 
metrics  listed  below  in  both  graphic  and 
tabular  form.  This  information  shall  be 
updated  as  necessary  for  the  current  reporting 
period.  The  metrics  data  shall  be  provided 
in  graphics  form  at  the  contract  level  for 
each  contractor/subcontractor.  Graphs  shall 
also  be  prepared  at  the  CSCI  level  for  certain 
metrics  as  designated  below.  Additional 
graphs  shall  be  provided  as  required  to 
address  high-risk  or  problem  CSCIs.  Graphs 
show  12  months  of  history  and  5  months  of 
projections  where  appropriate.  Revised  plans 
may  be  added  to  the  graphs,  but  previous 
plans,  including  the  original  plan,  shall  not 
be  removed. 

Software  Size 

For  the  current  reporting  period,  provide  in 


tabular  and  graphic  form  foi  each  CSCI  and 
for  each  source  code  language  an  estimate 
of: 

1 .  Newly  developed  Source  Lines  of  Code 
(SLOC); 

2.  Reused  SLOC  —  existing  code  used  as 
is; 

3.  Modified  SLOC  —  existing  code  requir¬ 
ing  change;  and 

4.  Total  SLOC  —  all  code  (sum  of  above). 

(Note:  SLOC  is  equal  to  the  count  of  all 
nonliteral  semicolons  (;)  in  each  Ada 
package.) 

Software  Personnel 

Provide  a  plan  for  each  contractor/ 
subcontractor  that  includes,  for  each 
month  of  the  contract: 

1.  The  planned  number  of  software 
personnel, 

2.  The  planned  number  of  Ada  software 
personnel,  and 

3.  The  planned  number  of  experienced  soft¬ 
ware  personnel. 

For  the  current  reporting  period,  provide  for 
each  contractor/subcontractor  actual  counts 
of: 

1 .  The  total  number  of  software  personnel, 

2.  The  number  of  Ada  software  personnel, 

3.  The  number  of  experienced  personnel, 
and 

4.  The  number  of  unplanned  personnel 
losses. 
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Software  personnel  include  engineering  and 
management  personnel  directly  involved 
with  software  system  planning,  requirements 
definition,  design,  coding,  test,  integration, 
documentation,  configuration  management, 
and  quality  assurance.  Experienced  personnel 
are  defined  as  those  with  over  5  years  of 
experience,  of  which  3  years  are  with  sys¬ 
tems  similar  to  the  one  under  development. 

Software  Volatility 

For  the  current  repotting  period,  provide: 

1 .  The  current  total  number  of  software 
requirements, 

2.  The  cumulative  number  of  requirements 
changes, 

3.  The  number  of  new  Software  Action 
Items  (SAIs), 

4.  The  cumulative  number  of  open  SAIs, 
and 

5.  The  number  of  SAIs  open  0-30  days, 
31-60  days,  61-90  days,  and  over  90 
days. 

An  SAI  is  defined  as  any  discrepancy,  clari¬ 
fication  item,  or  requirement  issue  that  must 
be  resolved  by  either  the  contractor  or  the 
Government. 

Computer  Resource  Utilization 

For  the  current  reporting  period,  provide 
estimated  percentage  utilizations  of: 

1 .  CPU  computation  power  for  each  CPU 
in  multi-CPU  configurations, 

2.  On-line  memory  for  each  CPU,  and 

3.  I/O  channel  capacity. 

Design  Complexity 

For  the  current  repotting  period,  provide: 

1 .  The  average  design  complexity  of  the  10 
peicent  most  complex  CSUs, 

2.  The  average  design  complexity  of  the  10 
percent  most  complex  CSCs, 


3.  The  design  complexity  of  each  CSCI, 
and 

4.  The  number  of  the  top  10  percent  most 
complex  CSUs  whose  design  complexity 
is  less  than  8,  8  to  20,  and  over  20. 

(Note:  See  the  attached  for  the  definition 
and  discussion  of  design  complexities.  (This 
discussion  is  contained  in  appendix  E  of  this 
document.)) 

Schedule  Progress 

Initially  provide  the  number  of  months  in 
the  program  schedule. 

For  the  current  reporting  period,  provide: 

1 .  The  budgeted  cost  of  work  performed 
(BCWP)  for  software, 

2.  The  budgeted  cost  of  work  scheduled 
(BCWS)  for  software,  and 

3.  The  number  of  months  in  the  program 
schedule  (if  revised). 

Design  Progress 

Provide  a  plan  that  includes,  for  each  month 
of  the  contract: 

1 .  The  number  of  System/Segment  Design 
Document  (SSDD)  software  requirements 
to  be  documented  in  Software  Require¬ 
ment  Specifications  (SRSs)  each  month, 

2.  The  number  of  SRS  requirements  to  be 
allocated  and  documented  as  CSCs  in 
Software  Design  Documents  (SDDs) 
each  month,  and 

3.  The  number  of  CSC  requirements  to  be 
allocated  and  documented  as  CSUs  in 
SDDs  each  month. 

For  the  current  reporting  period,  provide: 

1.  The  number  of  SSDD  software  require¬ 
ments  completely  documented  in  SRSs, 

2.  The  number  of  SRS  requirements  com¬ 
pletely  documented  as  CSCs  in  SDDs, 
and 

3.  The  number  of  CSC  requirements  com¬ 
pletely  documented  as  CSUs  in  SDDs. 


CSU  Development  Progress 

For  each  CSCI,  provide  a  plan  in  both 
graphic  and  tabular  form  that  includes,  for 
each  month  of  the  contract: 

1 .  The  number  of  CSUs  to  be  designed, 

2.  The  number  of  CSUs  to  be  tested,  and 

3.  The  number  of  CSUs  to  be  integrated. 

For  the  current  reporting  period,  provide  for 
each  CSCI  in  both  graphic  and  tabular  form 
actual  counts  of: 

1 .  The  number  of  CSUs  designed, 

2.  The  number  of  CSUs  tested,  and 

3.  The  number  of  CSUs  integrated. 

Testing  Progress 

Provide  a  test  plan  that  includes,  for  each 
month  of  the  contract: 

1 .  The  number  of  CSC  tests, 

2.  The  number  of  CSCI  tests,  and 

3.  The  number  of  system  tests  scheduled  to 
be  performed. 

For  the  cumnt  reporting  period,  provide: 

1.  The  cumulative  number  of  CSC,  CSCI, 
and  system  tests  passed; 

2.  1  he  number  of  new  software  problem 
reports  (SPRs); 


3.  The  cumulative  number  of  open  SPRs; 

4.  SPR  density,  i.e.,  the  cumulative  number 
of  SPRs  per  1000  SLOC; 

5.  The  number  of  SPRs  that  have  been  open 
0-30  days,  31-60  days,  61-90  days,  and 
over  90  days;  and 

6.  The  number  of  CSUs  whose  SPR  density 
is  0-10,  1 1-20,  21-30,  and  over  30. 

Incremental  Release  Content 

Provide  a  plan  that  includes: 

1.  The  number  of  CSUs  to  be  included  in 
each  incremental  build/release, 

2.  The  number  of  requirements  to  be  imple¬ 
mented  in  each  incremental  build/release, 
and 

3.  The  completion  date  for  each  incremental 
build/release. 

For  the  current  reporting  period,  provide: 

1 .  The  number  of  CSUs  that  were  added  to 
or  deleted  from  each  incremental  build/ 
release, 

2.  The  number  of  requirements  that  were 
added  to  or  deleted  from  each  incremental 
build/release,  and 

3.  Any  changes  in  completion  dates  for 
each  incremental  build/release. 


APPENDIX  D 

Glossary 


Spcdal  Terms 

SLOC 


New  SLOC 

Reused  SLOC 

Modified  SLOC 
Repotting  Period 


Source  Code 
Language 

Staff 


Target  Computer 
Resource 


Source  Lines  of  Code  —  includes  all  source  statements 
created  by  project  personnel  and  processed  into 
machine  code.  It  excludes  comments  and  unmodified 
utility  software.  It  includes  job  control  language,  for¬ 
mat  statements,  and  data  declarations.  It  also  includes 
newly  developed  support  software. 

SLOC  newly  developed,  that  is,  not  copied  from 
another  source. 

SLOC  copied  from  another  source,  then  reused  without 
change. 

SLOC  copied  from  another  source  and  modified. 

A  symbol  for  denoting  the  intervals  for  which  the  Soft¬ 
ware  Mi  aagement  Metrics  are  collected.  For  example, 
if  the  reporting  period  is  monthly,  the  reporting  period 
could  be  the  number  of  the  month  after  contract  award. 

The  language  used  during  software  development,  for 
example,  PL/I,  Ada,  Assembly,  and  so  on. 

All  those  directly  involved  in  the  software  development 
effort,  including  Project  Manager,  Project  Leader, 
Programmer,  Quality  Assurance  Staff,  and  Configura¬ 
tion  Management  Staff.  Also,  a  person  or  the  frac¬ 
tional  amount  of  a  person’s  time  devoted  to  the 
software  development  effort;  for  example,  a  staff 
member  who  is  devoting  only  half  time  to  this  program 
would  be  counted  as  (.5). 

The  computer  resource  to  be  used  in  the  delivered 
operational  system. 


Acronymns 

AI 

Artificial  Intelligence 

BCWP 

Budgeted  Cost  of  Work  Performed 

BCWS 

Budgeted  Cost  of  Work  Scheduled 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

COTS 

Commercial  Off-the-Shelf 

CPU 

Central  Processing  Unit 

CSC 

Computer  Software  Component 

CSCI 

Computer  Software  Configuration 
Item 

CSU 

Computer  Software  Unit 

DBMS 

Database  Management  System 

DID 

Data  Item  Description 

IFPP 

Instructions  for  Proposal 
Preparation 

IRC 

Incremental  Release  Content 

M 

Months 

MCCR 

Mission  Critical  Computer 
Resource 

MOTS 

Modified  Off-the-Shelf 

PCA 

Physical  Configuration  Audit 

PDL 

Program  Design  Language 

PDR 

Preliminary  Design  Review 

PMR 

Program  Management  Review 

QA 

Quality  Ass jrance 

SAI 

Software  Action  Item 

SDD 

Software  Design  Document 

SDR 

System  Design  Review 

SM 

Staff  Months 

SOW 

Statement  of  Work 

SPO 

System  Program  Office 

SPR 

Software  Problem  Report 

SRR 

System  Requirements  Review 

SRS 

Software  Requirements 
Specification 

SSDD 

System/Segment  Design  Document 

SSR 

Software  Specification  Review 

TIM 

Technical  Interchange  Meeting 

TRR 

Test  Readiness  Review 

V&V 

Validation  &  Verification 

WBS 

Work  Breakdown  Structure 

APPENDIX  E 


Design  Complexity  Definitions 
(See  References  2  and  4) 


Cydomatk  Complexity 

The  number  of  linearly  independent  paths  in 
a  CSU  that,  when  taken  in  combinatioi 
will  generate  every  possible  path.  Generally 
represented  as: 

C(U)  =  E  -  N  +  2  where  : 

E  is  the  number  of  connections  between 
nodes,  and 

N  is  the  number  of  nodes,  e.g.: 

C(U)  =  12  -  10  +  2  =  4 


In  the  above  example,  the  complexity  is 
calculated  to  be  4,  indicating  that  there  are 
four  basis  paths  that  will  generate  every 
possible  path: 

A-B-D-H-J 

A-C-G-I-J 

A-B-E-H-J 

A-C-F-I-J 

Basis  paths  for  testing  can  also  be  derived 
from  die  design  complexity  calculations  in 
the  following  paragraphs. 


CSU  Design  Complexity 

The  cyclomatic  complexity  of  a  CSU 
reducea  to  include  only  logic  that  interfaces 
with  other  CSUs  and  designated  as  DC(U). 
A  black  node  in  the  diagram  represents  an 
interface  with  another  CSU. 

DC(U)  =  7-  6  +  2  =  3 


CSC  Design  Complexity 

The  number  of  basis  paths  or  subtrees  that, 
when  taken  in  linear  combination,  will  gen¬ 
erate  the  entire  set  of  subtrees  for  a  CSC.  It 
is  equal  to  the  sum  of  CSU  design  complex¬ 
ities  minus  the  number  of  CSUs  plus  one. 

DC(CSC)  =  DC(Uj)  -  X  +  1,  where 

DC(Uj)  is  the  design  complexity  of  CSUj 
and  X  is  the  number  of  CSUs. 

DC(CSC)  =  6  -  4  +  1  =  3 
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CSCI  Design  Complexity 

The  number  of  basis  paths  or  subtrees  that, 
when  taken  in  linear  combination,  will  gen¬ 
erate  the  entire  set  of  subtrees  for  a  CSCI.  It 
is  equal  to  the  sum  of  CSC  design  complex¬ 
ities  minus  the  number  of  CSCs  plus  one. 

DC(CSCI)  =  DC(CSCj)  -  C  +  1, 
where 

DC(CSCj)  is  the  design  complexity  of 
CSCj  and  C  is  the  number  of  CSCs. 


